The
Exchange Transport Server architecture includes several areas whose
interdependence and configuration options you should understand.
1. Components of Message Transport
The Transport
Pipeline in Exchange Server 2010 consists of several components that
work together to route messages. There are four mechanisms to submit a
message into the transport pipeline:
SMTP Protocol
SMTP servers or messaging clients can submit mail using the standard
SMTP protocol. The message enters the transport pipeline using an SMTP
Receive connector.
Store Driver Mail from Mailbox Servers is submitted via Store Driver.
Pickup/Replay Folders Properly formatted messages can be dropped into the file system and the server will pick them up and process them.
Agent/System Generated The system can generate messages such as Non-Delivery Reports (NDRs) or Delivery Reports, and agents can generate messages such as Journal messages.
Figure 1 illustrates the Transport Pipeline and all components of a message transport to route messages through a Hub Transport server. As described later in the chapter, not all of these components are available for Edge Transport servers.
In the figure, the black
arrows refer to message delivery or receiving a message; the dotted
arrows illustrate how a message is submitted or sent.
1.1. Submission Queue
When the Microsoft Exchange Transport service starts, the categorizer creates one Submission queue on each Edge Transport server and Hub Transport server. The Submission queue stores all messages on disk until the categorizer processes them for further delivery.
The categorizer cannot process a message unless a server promotes it to
the Submission queue. While the categorizer processes a message, it
remains in the Submission queue. After successful categorization, the
message is removed from the submission queue.
Charlie Chung
Program Manager, Microsoft Corp, Redmond
What's useful about knowing
that the submission queue is behind the categorizer is that if the
submission queue is growing, you need to look at categorizer to figure
out why messages are backing up. For example, most antivirus products
are registered on a categorizer event. If that agent is misbehaving,
the submission queue will grow.
|
On the Hub
Transport server, message submission can occur through the store
driver, through the pickup or replay directory, directly by an agent,
or by the Receive connector. On the Edge Transport server, submission
is generally only through the Receive connector because messages flow
in from the Internet, although pickup or replay directories exist.
1.2. Delivery Queue
Delivery
queues hold messages before they are delivered to their target
database. Depending on where they should route the messages to, they
are called mailbox delivery queue and remote delivery queue.
Mailbox delivery queues hold messages that are being delivered to a mailbox server located in the same site by using encrypted Exchange RPC. Mailbox delivery queues, one queue for each database, exist on Hub Transport servers only.
Remote
delivery queues hold messages that are being delivered to a remote
server by using SMTP. Remote delivery queues can exist on both Hub
Transport servers and Edge Transport servers, and more than one remote
delivery queue can exist on each server. On an Edge Transport server,
these destinations are external SMTP domains or SMTP connectors. On a
Hub Transport server, these destinations are SMTP connectors that are
outside the Active Directory site in which the Hub Transport server is
located or a remote SMTP connector.
1.3. Categorizer
The categorizer retrieves one message at a time from the Submission queue, and always picks the oldest message first. On Hub Transport
servers, the categorizer completes the following three steps: recipient
resolution, routing resolution, and content conversation. On Edge
Transport servers the categorizer puts the message into the delivery
queue and routes it to a Hub Transport server.
1.4. Store Driver
Messages sent from mailboxes enter the transport pipeline from the sender's Outbox. After the message is put in the sender's Outbox, the store driver is alerted by the Microsoft
Exchange Mail Submission service, the store driver retrieves the
message from the sender's Outbox, and then puts it into the Submission
queue on a Hub Transport server in the same Active Directory site of
the Mailbox server.
The store driver is
also responsible for picking up messages that should be delivered
locally (delivered in the same Active Directory site) from a delivery
queue. The store driver puts the message in the recipient's inbox on
the respective Mailbox server.
It is important to understand
that store driver is an interface between the Hub server and the
Mailbox server. That is why you see it in the beginning and the end of
the Transport Pipeline—it is used to for messages submitted by the
Mailbox server and to deliver messages to the target mailbox database.
1.5. Microsoft Exchange Mail Submission Service
The Microsoft Exchange Mail
Submission service is a notification service running on Mailbox
servers. It notifies the store driver on a Hub Transport server in the
local Active Directory site when a message is available for retrieval
from a sender's Outbox.
1.6. Pickup and Replay Directory
The Pickup and
Replay directories serve basically the same purpose: to process
properly formatted text files usually generated by applications into
messages. The pickup
directory is used by applications or users; the Replay directory is
used to resubmit exported Exchange messages and to receive messages
from foreign gateway servers. Because their purpose is almost
identical, this section only focuses on the Pickup directory. Details
about the Replay directory can be found at http://technet.microsoft.com/en-us/library/bb124230.aspx.
The Pickup directory is another way for messages to enter the message
transport pipeline by being placed into the Pickup directory on a Hub
Transport server or an Edge Transport server. The Pickup directory is
by default located under <Exchange_Installation_Path>\TransportRoles\Pickup. It can be modified using the following cmdlet:
Set-TransportServer <Servername> -PickupDirectoryPath "C:\Pickup"
The Pickup directory is used to support legacy applications that are not capable of sending messages using MAPI or SMTP protocol but instead by using a simple text file (extension .eml) that is stored in the Pickup directory.
The text file has to
include a certain formatting so that Exchange can compose a message out
of the text file and sent it. The following requirements must be met:
The message file
must be a text file that complies with the basic SMTP message format.
MIME message header fields and content are supported.
Only one e-mail address can exist in the Sender field. Multiple e-mail addresses aren't allowed.
At least one e-mail address must exist in the To, Cc, or Bcc fields.
A blank line must exist between the message header and the message body.
The following is an example of a valid message that can be processed by the Pickup directory:
To: [email protected]
From: [email protected]
Subject: Pickup Directory Test
Hi!
This is a Pickup Directory Test
Sigi
The pickup process is quite simple: you store a message as a <message>.eml file in the directory, the Pickup directory is checked every five seconds (this cannot be changed), and then .eml files will be processed.
The processing of messages in the pickup directory always follows the same scheme:
Rename <message>.eml to <message>.tmp. (If the file exists, the date and time are added to the name as well.)
If the <message>.eml did not include the required formatting, the file is renamed to <message>.bad.
After
the message is successfully queued for delivery, a close command is
issued, and the .tmp file is deleted from the Pickup directory.
Note: If the Microsoft Exchange
Transport service is restarted when there are .tmp files in the Pickup
or Replay directories, all .tmp files are renamed as .eml files and are
reprocessed. This could lead to duplicate message transmission.
2. Message Queues on Transport Servers
Every message that is sent between Transport servers is temporary stored on the Transport server in a place called a message queue. In this location, the message waits for the next step in processing. As explained in Table 1, six different message queues are available on a Transport server. You will find more details on the specific queues later in this chapter.
Table 1. Message Queues in Exchange 2010
QUEUE NAME | DESCRIPTION |
---|
Mailbox delivery queue (or MAPI delivery) | Delivers
messages to mailbox databases located in the same Active Directory
site. One queue is available per database. This queue only exists on
Hub Transport servers. |
Remote delivery queue | On
Hub Transports, delivers messages to a Hub Transport server in a remote
Active Directory site. On Edge Transport servers, delivers messages to
remote SMTP domains. |
Submission queue | Stores messages that are then processed by the categorizer for further delivery. |
Shadow redundancy queue | Copies
of messages that are sent to a remote Hub Transport that did not yet
report successful delivery of the message are stored here. |
Poison message queue | Stores isolated messages that are detected to be potentially harmful to the Exchange 2010 system. |
Unreachable queue | Contains
messages that can't be routed to their destinations, most likely
because of Active Directory replication latency or other configuration
issues. |
Exchange 2010 stores all queues
in the queue database, which is described in the next section. The
queues itself can be managed using either the Queue Viewer or using the
Get-Queue cmdlet, as shown in Figure 2.
3. Queue Database
All message queues are stored in a single database called a queue database. The queue
database is based on the Extensible Storage Engine (ESE) database
technology, which is also used by the Mailbox Server. The queue
database is made up of a database file and several log files to record
transactions. The checkpoint file tracks which transaction log files
have been committed to the database. During a service shutdown of the
Microsoft Exchange Transport service, all transaction log files are always committed to the database.
The queue database uses circular
logging, which means that the history of committed transactions that
are stored in the transaction log files is not maintained. Any
transaction log file older than the current checkpoint is immediately
and automatically deleted and thus cannot be used to replay queue database recovery.
3.1. Queue Database Files
The queue database consists of several files, which are stored in the following default location: <Exchange_Install_Path>\TransportRoles\data\Queue. Table 2 queue database files and their purpose. provides an overview of the
Table 2. Queue Database Files and Purpose
FILE | DESCRIPTION |
---|
Mail.que | This file is the queue database and stores all queued messages after they are committed from the transaction log files. |
Tmp.edb | This temporary database file is used to verify the queue database schema on startup. |
Trn*.log | This
transaction log records all changes to the queue database. Changes to
the database are first written to the transaction log and are then
committed to the database. Trn.log is the current active transaction
log file. Trntmp.log is the next provisioned transaction log file that
is created in advance. If the existing Trn.log transaction log file
reaches its maximum size of 5 MB, Trn.log is renamed to TrnXXXX.log, where XXXX is a sequence number. Trntmp.log is then renamed Trn.log and becomes the current active transaction log file. |
Trn.chk | The
checkpoint file. It tracks the transaction log entries that have been
committed to the database. This file is always in the same location as
the mail.que file. |
Trnres00001.jrs Trnres00002.jrs | These files are used to reserve emergency storage if the transaction log volume becomes full. |
3.2. Queue Database Configuration Options
Several configuration options exist for the queue database. Unlike Mailbox or public folder databases, settings for the queue
database cannot be configured in the Exchange Management Console or
with EMS; instead, you need to change the settings in the EdgeTransport.exe.config file located at <Exchange_Server_Install>\Bin. If you change any settings, you need to restart the Microsoft Exchange Transport service for the changes to take effect.
Table 3 provides an overview of options available to configure the queue database.
Table 3. Overview of Parameters to Configure the Queue Database
PARAMETER | DESCRIPTION |
---|
QueueDatabasePath | Directory of the queue
database files. If you change the directory, make sure that the new
directory exists or use the Move- TransportDatabase.ps1 script. |
QueueDatabaseLoggingPath | Directory of the queue
database log files. If you change the directory, make sure that the new
directory exists or use the Move-TransportDatabase.ps1 script. |
QueueDatabaseBatchSize | Defines the number of database I/O operations that can be grouped together before they are executed. Default: 40. |
QueueDatabaseBatchTimeout | Defines
the maximum time in milliseconds that the database will wait for
multiple database I/O operations to group. Default: 100. |
QueueDatabaseMaxConnections | Defines the number of ESE database connections that can be open. Default: 4. |
QueueDatabaseLoggingBufferSize | Specifies
the memory that is used to cache the transaction records before they
are written to the transaction log file. Default: 5,242,880 bytes. |
QueueDatabaseLoggingFileSize | Defines the maximum size of a transaction log file before a new log file is opened. Default: 5,242,880 bytes. |
QueueDatabaseMaxBackgroundCleanupTasks | Defines the maximum number of background cleanup work items that can be queued to the database engine thread pool. Default: 32. |
QueueDatabaseOnlineDefragEnabled | Enables or disables scheduled online defragmentation. Default: $true. |
QueueDatabaseOnlineDefragSchedule | Defines the time to start the online defragmentation. Default: 1:00:00 (1:00 A.M.). |
QueueDatabaseOnlineDefragTimeToRun | Specifies
the maximum time the online defragmentation task is allowed to run. If
the defragmentation task is not finished in time, the queue database is left in a consistent state. Default: 3:00:00 (3 hours). |
To change the folder or drive of the queue database or log files, you can use the use the Move-TransportDatabase.ps1 script available in the <Exchange_Server_Install>\Scripts folder. For example, if you want to move the queue database and log files to the D:\Queue folder, you use the .\Move-TransportDatabase.ps1 –QueueDatabasePath D:\Queue –QueueDatabaseLoggingPath D:\Queue command.